Papers with -shot prompting
Measuring Linguistic Competence of LLMs on Indigenous Languages of the Americas (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for linguistic knowledge of Indigenous languages of the Americas focus on high- and medium-resource languages with substantial digital presence. |
| Approach: | They propose a framework for probing large language models’ linguistic knowledge of Indigenous languages of the Americas using zero-shot prompting and few-shot probing. |
| Outcome: | The proposed framework evaluates models from five major families on 13 Indigenous languages including Bribri, Guarani, and Nahuatl. |
A Comparative Analysis of Conversational Large Language Models in Knowledge-Based Text Generation (2024.eacl-short)
Copied to clipboard
| Challenge: | Generating natural language text from graph-structured data is essential for conversational information seeking. |
| Approach: | They conduct an empirical analysis of conversational large language models in generating natural language text from semantic triples using a WebNLG dataset. |
| Outcome: | The proposed models improve their ability to generate natural language text from semantic triples using few-shot prompting, post-processing, and efficient fine-tuning techniques. |
Can a Large Language Model Keep My Secrets? A Study on LLM-Controlled Agents (2025.acl-srw)
Copied to clipboard
| Challenge: | Using large language models, agents can assist with natural language tasks when given access to confidential data. |
| Approach: | They created a synthetic dataset consisting of confidentiality-aware planning and deduction tasks in organizational access control. |
| Outcome: | The proposed model can perform tasks similar to humans when given access to confidential data. |
GlyphPattern: An Abstract Pattern Recognition for Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for abstract pattern recognition are easier because they do not involve a natural language description of the pattern. |
| Approach: | They present a dataset that pairs human-written descriptions of visual patterns with three visual presentation styles. |
| Outcome: | The proposed benchmark pairs human-written and human-verified patterns with three visual presentation styles. |
TrojFSP: Trojan Insertion in Few-shot Prompt Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Prompt tuning on a few data samples presents security issues, e.g., Trojan attacks. |
| Approach: | They propose a method to transfer established data poisoning attacks directly to few-shot prompt tuning, a technique to address the poisoned imbalance issue. |
| Outcome: | The proposed method achieves an ASR of over 99% while maintaining negligible decreases in CDA. |
Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification (2026.acl-srw)
Copied to clipboard
| Challenge: | idiomatic light verb constructions (LVCs) are challenging for multiword expression processing . they share the same surface form as fully literal verb–object combinations . |
| Approach: | They frame Turkish LVC detection as a binary classification task . they compare a supervised Turkish encoder baseline to three instruction-tuned LLMs . |
| Outcome: | The proposed method improves Turkish LVC detection on a controlled set with matched negatives and positives. |
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on explainable NLP has shown that few-shot prompting can enable large pre-trained language models (LLMs) to generate grammatical and factual explanations for data labels. |
| Approach: | They propose to use few-shot prompting to generate grammatical and factual explanations for data labels by varying the hardness of the test samples and in-context samples to investigate the link between explainability and sample hardness. |
| Outcome: | The proposed model can generate grammatical and factual explanations for data labels with few-shot prompting. |
Bootstrapping Multilingual Semantic Parsers using Large Language Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Despite cross-lingual generalization, translation models require significant amounts of labeled data for many low-resource languages . brittle translation services may be due to domain mismatch between input text and general-purpose text . |
| Approach: | They propose to use large language models to translate English datasets into several languages via few-shot prompting. |
| Outcome: | The proposed method outperforms a strong translation-train baseline on 41 out of 50 languages. |
Can LLMs Clarify? Investigation and Enhancement of Large Language Models on Argument Claim Optimization (2025.coling-main)
Copied to clipboard
| Challenge: | While Large Language Models (LLMs) have demonstrated proficiency in text rewriting tasks such as style transfer and query rewrite, their application to claim optimization remains unexplored. |
| Approach: | They propose to use a sliding window mechanism to evaluate the performance of large language models in claim clarification tasks under different settings. |
| Outcome: | The proposed model improves the performance of three LLMs on the claim clarification task under zero-shot, few-shot and supervised fine-tuning settings. |
Role-Conditioned Refusals: Evaluating Access Control Reasoning in Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) blur role boundaries by producing unrestricted responses. |
| Approach: | They propose to extend the Spider and BIRD text-to-SQL datasets with real-time PostgreSQl role-based policies at the table and column levels. |
| Outcome: | The proposed model improves refusal precision and lowers false permits. |
SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models are far from achieving the robustness and consistency required for safe biomedical NLI applications. |
| Approach: | They propose a framework that leverages external ontologies to instantiate diverse syllogistic arguments for biomedical NLI by identifying valid conclusions and extracting supporting evidence. |
| Outcome: | The proposed framework evaluates large language models on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. |
Understanding and Mitigating Language Confusion in LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language. |
| Approach: | They develop a language confusion benchmark to evaluate LLMs' inability to consistently generate text in a user’s desired language. |
| Outcome: | The proposed model fails to consistently respond in the correct language, despite being prone to high temperatures and complex prompts. |
Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Using a Llama-3.2-3B-Instruct backbone, we evaluate four adaptation mechanisms across four benchmarks: Gorilla APIBench, Spider 2.0, WebArena, and InterCode. |
| Approach: | They compare hypernetwork-based LoRA adaptation against carefully designed few-shot prompting in a controlled experiment . they find that few- shot prompting contributes +21.5% to performance and documentation contributes 0% . |
| Outcome: | The hypernetwork-based LoRA adaptation provides no measurable improvement over few-shot prompting alone. |
Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for knowledge graph creation (KGC) are limited in their ability to scale up to text common in many real-world applications. |
| Approach: | They propose a framework for knowledge graph creation from input text using a pre-defined schema and a trained component that retrieves schema elements relevant to the input text. |
| Outcome: | The proposed framework extract-define-canonicalize extracts high-quality triplets with a succinct self-generated schema without any parameter tuning and with significantly larger schemas compared to prior works. |
CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models (2024.acl-long)
Copied to clipboard
Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua
| Challenge: | Large language models are used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown. |
| Approach: | They propose a benchmark for evaluating large language models using a well-organized taxonomy. |
| Outcome: | The proposed model is based on a well-organized taxonomy and compares it with other models. |
ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation resources lack domain and language diversity, limiting the ability for cross-domain and cross-lingual analyses. |
| Approach: | They propose to use a multilingual multi-domain dataset to benchmark multilingual and monolingual models for multilingual readability assessment. |
| Outcome: | The proposed model trains better in supervised, unsupervised, and few-shot prompting settings and identifies shortcomings in state-of-the-art unsupervised methods. |
Prompting open-source and commercial language models for grammatical error correction of English learner text (2024.findings-acl)
Copied to clipboard
Christopher Davis, Andrew Caines, O Andersen, Shiva Taslimipoor, Helen Yannakoudakis, Zheng Yuan, Christopher Bryant, Marek Rei, Paula Buttery
| Challenge: | Recent advances in generative AI have enabled us to prompt large language models (LLMs) to produce texts which are fluent and grammatical. |
| Approach: | They evaluate model performance by measuring their performance on established benchmarks. |
| Outcome: | The proposed models outperform supervised English GEC models on fluency correction benchmarks and commercial LLMs on edit benchmarks. |
Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning (2023.findings-emnlp)
Copied to clipboard
Duarte Alves, Nuno Guerreiro, João Alves, José Pombal, Ricardo Rei, José de Souza, Pierre Colombo, Andre Martins
| Challenge: | Large language models (LLMs) are a promising avenue for machine translation (MT) however, their effectiveness depends on the choice of few-shot examples and they often require extra post-processing due to overgeneration. |
| Approach: | They propose a method that incorporates few-shot examples during finetuning to improve performance on MT tasks. |
| Outcome: | The proposed method outperforms few-shot prompting while eliminating the need for in-context examples. |
Multilingual Large Language Models Are Not (Yet) Code-Switchers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual Large Language Models are not specifically trained with objectives for managing code-switching scenarios. |
| Approach: | They propose to use multilingual Large Language Models to perform sentiment analysis, machine translation, summarization and word-level language identification to compare their performance to fine-tuned models of much smaller scales. |
| Outcome: | The proposed models show that they underperform in comparison to fine-tuned models of much smaller scales. |
Automated Knowledge Graph Construction using Large Language Models and Sentence Complexity Modelling (2025.emnlp-main)
Copied to clipboard
Sydney Anuyah, Mehedi Mahmud Kaushik, Sri Rama Krishna Reddy Dwarampudi, Rakesh Shiradkar, Arjan Durresi, Sunandan Chakraborty
| Challenge: | integrating coreference and decomposition increases recall on rare relations by over 20%. |
| Approach: | They propose an open-source pipeline for extracting sentence-level knowledge graphs by combining robust coreference resolution with syntactic sentence decomposition. |
| Outcome: | The proposed pipeline achieves a 99.8% exact-match accuracy on sentence simplification. |
Dunhuang-Bench: How Well Do MLLMs Understand Cultural Heritage? (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Multimodal Large Language Models (MLLMs) have led to extensive evaluations on Chinese cultural benchmarks. |
| Approach: | They construct a large-scale benchmark comprising 486 images and 22,970 QA pairs to evaluate MLLMs' cultural understanding. |
| Outcome: | The proposed benchmark incorporates three task formats to evaluate MLLMs’ cultural understanding: Question Answering with Text Description, Multi-turn Dialogue, and Question Answers with Choices. |
Anecdoctoring: Automated Red-Teaming Across Language and Place (2025.emnlp-main)
Copied to clipboard
| Challenge: | Disinformation is among the top risks of generative AI misuse . red-teaming datasets are typically US- and English-centric . |
| Approach: | They propose a red-teaming approach that generates adversarial prompts across languages and cultures by clustering misinformation claims into broader narratives and enhancing an attacker LLM. |
| Outcome: | The proposed approach produces higher attack success rates and interpretability benefits relative to few-shot prompting. |
P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains (2024.findings-emnlp)
Copied to clipboard
Simeng Han, Aaron Yu, Rui Shen, Zhenting Qi, Martin Riddell, Wenfei Zhou, Yujie Qiao, Yilun Zhao, Semih Yavuz, Ye Liu, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Dragomir Radev, Rex Ying, Arman Cohan
| Challenge: | Existing methods on understanding the capabilities of LLMs in logical reasoning rely on binary entailment classification or synthetically derived rationales. |
| Approach: | They propose to annotate a human-annotated dataset consisting of diverse and complex reasoning chains for a set of realistic logical reasoning stories also written by humans. |
| Outcome: | The proposed model outperforms existing methods on understanding the capabilities of LLMs in logical reasoning by 10% or more. |
LangMark: A Multilingual Dataset for Automatic Post-Editing (2025.acl-long)
Copied to clipboard
Diego Velazquez, Mikaela Grace, Konstantinos Karageorgos, Lawrence Carin, Aaron Schliem, Dimitrios Zaikis, Roger Wechsler
| Challenge: | Automated post-editing (APE) aims to correct errors in machine-translated text . lack of large-scale multilingual datasets specifically tailored to NMT outputs hinders APE development . |
| Approach: | They propose to use a human-annotated multilingual APE dataset for English translation to seven languages to address this gap. |
| Outcome: | The proposed dataset offers both linguistic diversity and scale. |